Interactions of virtual objects with surfaces

ABSTRACT

Embodiments are disclosed that relate to operating a user interface on an augmented reality computing device comprising a display system. For example, one disclosed embodiment includes displaying a virtual object via the display system as free-floating, detecting a trigger to display the object as attached to a surface, and, in response to the trigger, displaying the virtual object as attached to the surface via the display system. The method may further include detecting a trigger to detach the virtual object from the surface and, in response to the trigger to detach the virtual object from the surface, detaching the virtual object from the surface and displaying the virtual object as free-floating.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/005,530 filed Jan. 25, 2016, which is a continuation of U.S.patent application Ser. No. 13/893,146 filed May 13, 2013, now U.S. Pat.No. 9,245,388, the entire contents of each of which are incorporatedherein by reference for all purposes.

BACKGROUND

An augmented reality computing device may display virtual or holographicobjects overlaid on an environment that includes real objects andreal-world surfaces, e.g., walls, tables, etc. For example, ahead-mounted display device may include a see-through display systemconfigured to display images yet not obscure a real-world backgroundviewable through the see-through display system.

SUMMARY

Embodiments for operating a user interface on an augmented realitycomputing device comprising a display system are disclosed. For example,one disclosed embodiment provides a method including displaying avirtual object via the display system as free-floating, detecting atrigger to display the object as attached to a surface, and, in responseto the trigger, displaying the virtual object as attached to the surfacevia the display system. The method may further include detecting atrigger to detach the virtual object from the surface, and, in responseto the trigger to detach the virtual object from the surface, detachingthe virtual object from the surface and displaying the virtual object asfree-floating.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used to limit the scope of the claimed subject matter. Furthermore,the claimed subject matter is not limited to implementations that solveany or all disadvantages noted in any part of this disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B show a schematic depiction of an example use environmentfor an augmented reality computing system.

FIG. 2 schematically shows an example embodiment of an augmented realitydisplay device.

FIG. 3 is a block diagram of the display device of FIG. 2.

FIGS. 4-10 show example interactions of an augmented reality displaydevice user with virtual objects in an augmented reality environment inaccordance with the disclosure.

FIG. 11 shows an example method for operating a user interface on anaugmented reality computing device in accordance with the disclosure.

FIG. 12 schematically shows an example computing system.

DETAILED DESCRIPTION

A see-through display system, e.g., a head-mounted display (HMD) deviceor other near-eye display device, may be used to present an augmentedreality view of a background scene to a user. Such an augmented realityenvironment may include various virtual and real objects with which auser may interact via user inputs, such as speech inputs, gestureinputs, eye tracking inputs, motion inputs, and/or any other suitableinput type.

Certain types of virtual objects may be desired by a user of asee-through display system to remain in view of the user withoutbecoming too intrusive in the user's view. For example, a user of anaugmented reality computing device may desire to have a floatingholographic television follow the user around while they perform varioustasks in an environment. Other example virtual objects that may bepersistently displayed to a user in their field of view may include afloating holographic robot assistant or control panel interface whichenables the user to change properties of the environment, e.g., tochange a color of a room.

In order to keep such virtual objects in persistent view of a user whilereducing intrusiveness of the virtual object on a user's field of view,in some examples, persistent virtual objects may be attached toreal-world surfaces in the environment, e.g., attached to walls, so thatthe virtual object may float across the surface of the wall. Forexample, while displayed on a surface, a rotation of the virtual objectmay be aligned with the surface and may slide across the surface to acomfortable viewing position for the user. Similarly, virtual objectsalso may be attached to virtual surfaces, such as a virtual wall orother virtual environment structures.

However, various issues may arise in managing interactions of suchpersistent virtual objects with real-world surfaces in an augmentedreality environment. For example, it may be desirable to permit usercontrol of when and how virtual objects are attached to real-worldsurfaces. Further, the attachment of virtual objects to real-worldsurfaces may depend on a type and state of the virtual object andcharacteristics of the real-world surfaces. Additionally, as apersistent virtual object moves about a real-world environment, variousenvironmental challenges may be present in transitioning the virtualobject from a free-floating state to a surface-attached state and intransitioning the virtual object from one location to another in theenvironment.

Accordingly, embodiments are disclosed herein that relate tofacilitating movement of persistent virtual objects through an augmentedreality environment and managing attachments of virtual objects withreal-world surfaces in the augmented reality environment. Briefly, thedisclosed embodiments relate to displaying a virtual object on anaugmented reality display device as free-floating, and, in response to atrigger to display the object as attached to a surface (real orvirtual), displaying the virtual object as attached to the surface. Thetrigger may comprise any suitable condition or event, including but notlimited to a user input and a threshold distance between the virtualobject and the real-world surface. While embodiments presented below aredescribed in the context of real-world surfaces, it will be understoodthe illustrated concepts may be used with any suitable surfaces, real orvirtual. Likewise, while embodiments are described in the context of asee-through head-mounted display system, it will be understood that anyother suitable type of augmented reality display system may be used.Examples include, but are not limited to, a portable device (e.g. tabletcomputer, smart phone, laptop computer, etc.) having a camera and adisplay configured to display image data acquired by the camera, whereinaugmented reality images may be displayed on the display in the form ofvirtual objects composited with real-world image data acquired by theimage sensor.

FIGS. 1A and 1B shows an example embodiment of a use environment for anembodiment of a head-mounted display device 104, wherein environment 100takes the form of a living room. It should be understood thatenvironment 100 is presented for the purpose of example, and that a useenvironment may take any other suitable form.

A user 106 is viewing the living room through an augmented realitycomputing device in the form of the see-through head-mounted displaydevice 104, and may interact with the augmented environment via a userinterface of display device 104. FIGS. 1A and 1B also depict a userfield of view 102 comprising a portion of the environment viewablethrough the display device 104, and thus the portion of the environmentthat may be augmented with virtual images displayed via the displaydevice 104. In some embodiments, the user field of view 102 may besubstantially coextensive with the user's actual field of vision, whilein other embodiments the user field of view 102 may occupy a lesserportion of the user's actual field of vision.

The augmented environment may include a plurality of real-world surfacesand real-world objects. For example, as shown in FIGS. 1A and 1B, theenvironment includes a couch 114, a table 116, a first wall or surface118, a second wall 120, and a third wall 122. Further, the real-worldsurfaces in environment 100 may include various surface discontinuities,such as corners, doors, windows, real-world objects coupled to oradjacent to the surfaces, etc. Example surface discontinuities shown inFIGS. 1A and 1B include corner or discontinuity 124 at the interface ofwall 120 with surface 118, corner 126 at the interface of surface 118and wall 122, and a door 128 in wall 122. It should be understood thatthe example real-world surfaces shown in FIGS. 1A and 1B arenon-limiting examples. Other example real-world surfaces may includeexterior surfaces, e.g., a side of a building, glass shop fronts insidea shopping mall, and any other suitable surface.

As will be described in greater detail below, display device 104 maycomprise one or more outwardly facing image sensors (e.g.,two-dimensional cameras and/or depth cameras) configured to acquireimage data (e.g. color/grayscale images, depth images/point cloud data,etc.) representing user environment 100 as the user navigates theenvironment. This image data may be used to obtain information regardingthe layout of the environment (e.g., three-dimensional surface map,etc.) and objects contained therein, such as couch 114, table 116, wall120, surface 118, wall 122, door 128, and corners or discontinuities 126and 124. Further, the image data may be used to obtain information aboutcharacteristics or attributes of real-world surfaces in the environment.Example characteristics or attributes of real-world surfaces may includecolor, texture, shape, size, etc. As described in more detail below,such surface characteristics or attributes may be used in policies whichspecify how virtual objects interact with the real-world surfaces in theenvironment.

As remarked above, in some examples one or more virtual objects may bedisplayed to user 106 via see-through display device 104 so that the oneor more virtual objects remain in the field of view 102 of user 106 asthe user moves about or looks around the environment. For example, FIG.1A shows an example virtual object 130 displayed as free-floating inenvironment 100 via see-through display device 104 to user 106. Forexample, free-floating virtual object 130 may be a free-floatingholographic television which moves about the environment based on alocation of user 106 within the environment 100 and/or based on alocation of a gaze or focus 132 of the user 106 in the environment.Free-floating virtual object 130 may be any suitable virtual object,e.g., a planar objects such a floating virtual television or homeautomation control panel, or a non-planar 3-D virtual object such asflowers, plants, statues, art, etc.

FIG. 1A shows free-floating virtual object 130 displayed at a locationof user focus 132 in environment 100. In some examples, thefree-floating virtual object 130 may be persistently displayed at thelocation of user focus 132 even when the location of user focus changes.For example, if user 106 changes the location of focus 132 to a newlocation within environment 100 then virtual object 130 may be displayedat the new location in the environment so that the virtual objectremains at a position in the environment where the user is gazing. Forexample, as described in more detail below, display device 104 mayinclude various sensors to determine a location of user gaze or focus inthe environment and/or orientation and/or posture of the user in theenvironment so that display of the free-floating virtual object may beadjusted accordingly. For example, a rotation of the free-floatingobject may be adjusted based on an orientation or body posture of user106 in the environment.

In some examples, user 106 may desire to attach a free-floating virtualobject, e.g., free-floating virtual object 130 shown in FIG. 1A, to areal-world surface in environment 100. For example, as shown in FIG. 1B,virtual object may be attached to real-world wall or surface 118 so thatthe virtual object does not obstruct the user's view of other objects inthe environment. As another example, virtual object may be attached to atop surface of table 116, or attached to a position on couch 114, orattached to other suitable surfaces in environment 100.

Free-floating virtual object displayed in environment 100 may beattached and detached from a real-world surface in any suitable manner.For example, as shown in FIGS. 1A and 1B, user 106 may cause thefree-floating virtual object 130 shown in FIG. 1A to move towardssurface 118 by changing the position of user focus 132 from a firstposition shown in FIG. 1A to a second position of user focus 132 at alocation proximate to surface 118 as shown in FIG. 1B, potentially incombination with another trigger. As described in more detail below,various triggers may be used to cause a virtual object to be displayedas attached to a real-world surface. For example, after moving thevirtual object 130 to a location proximate to surface 118 by changing alocation of gaze, the user may provide a voice command or speech inputto attach the virtual object to surface 118. As another example, virtualobject 130 may automatically be displayed as attached to surface 118when certain environmental conditions are met, e.g., when a distancebetween the virtual object 130 and surface 118, or user 106 and surface118, is less than a threshold distance. Further, as described in moredetail below, various triggers may be used to detach a virtual objectfrom a real-world surface or to move an attached virtual object alongthe real-world surface to which it is attached. For example, after avirtual object is attached to a real-world surface, in some examples,the virtual object may move along the real-world surface as the userchanges position and/or gaze direction within the environment. However,in other examples, a virtual object attached to a real-world surface maybe pinned to the real world surface so that the virtual object isdisplayed at a fixed location on the real-world surface.

In some examples, one or more policies may be obtained and used tomanage attachment of persistent virtual objects with real-worldsurfaces. For example, policies may dictate one or more of whichreal-world surfaces the virtual object may be attached to and under whatconditions the virtual object may be attached to a real-world surface.The virtual object may then be displayed on the see-through displaysystem based on the one or more policies associated with the virtualobject. In some examples, these policies may be specified at a time ofcreation of the object so that the policies are built into the object.In other examples, these policies may be based on user input to allowusers to bias certain areas and specific holographic objects which wouldobtain or refrain from receiving specific characteristics.

For example, such policies may specify whether the virtual object can beattached to a particular real-world surface and/or whether the virtualobject is pin-able to the real-world surface, e.g., whether or not thevirtual object can be attached at a fixed location on the real-worldsurface. Further, such policies may depend on a type and/or state of thevirtual object. For example, depending on a type and/or state of anobject, the object may be permitted to attach to certain types ofreal-world surfaces and not permitted to attach to other types ofreal-world surfaces.

Further, these policies may specify how the attachment between a virtualobject and a real-world surface is displayed. These policies may dependon various attributes of the real-world surfaces, e.g., as detected viaone or more sensors in display device 104. For example, attributes ofreal-world surfaces such as a type of surface, e.g., “wall” or “table,”surface transparency, e.g., translucent or opaque, surface texture,e.g., wood or glass, etc. may be determined and referenced in a policyassociated with a virtual object to dictate how the virtual object isdisplayed relative to the real-world surface.

For example, if the virtual object is a floating holographic television,then the virtual object may be permitted to attach to an unobstructedarea of a wall at a specified height, e.g., higher than five feet orsome other height threshold, or may be permitted to attach to a tabletop but may not be permitted to attach to a couch in the environment. Asanother example, a virtual chess board object may be permitted to snapto a table surface but not be attached to a wall. As another example, avirtual poster may be permitted to snap to an unobstructed area of awall but not onto a surface of a table. Further, the virtual poster maybe permitted to only attach to a wall with a color that matches thevirtual poster. As another example, a virtual television may bedisplayed as attached to a surface of a table at a bottom edge of thevirtual television whereas when attached to a wall, the virtualtelevision may be displayed with its back abutting the wall. As stillanother example, policies may specify padding space around a virtualobject. For example, certain types of virtual objects may be displayedwith a small amount of padding space around the object so that thevirtual object is permitted to be displayed in a tight fit on a surfacerelative to surrounding surface features whereas other types of virtualobjects may only be permitted to be displayed on surfaces which providea large amount of padding space around the object.

A see-through display device according to the present disclosure maytake any suitable form, including but not limited to near-eye devicessuch as the head-mounted display device 104 of FIGS. 1A and 1B. FIG. 2shows an example embodiment of a see-through display system 300, andFIG. 3 shows a block diagram of the display system 300.

Display system 300 comprises one or more lenses 302 that form a part ofa display subsystem 304, such that images may be displayed via lenses302 (e.g. via projection onto lenses 302, waveguide system(s)incorporated into lenses 302, and/or in any other suitable manner).Display system 300 further comprises one or more outward-facing imagesensors 306 configured to acquire images of a background scene and/orphysical space being viewed by a user, and may include one or moremicrophones 308 configured to detect sounds, such as voice commands froma user. Outward-facing image sensors 306 may include one or more depthsensors and/or one or more two-dimensional image sensors.

Display system 300 may further comprise a gaze detection subsystem 310configured to detect a direction of gaze of each eye of a user or adirection or location of focus, as described above. Gaze detectionsubsystem 310 may be configured to determine gaze directions of each ofa user's eyes in any suitable manner. For example, in the depictedembodiment, gaze detection subsystem 310 comprises one or more glintsources 312, such as infrared light sources, configured to cause a glintof light to reflect from each eyeball of a user, and one or more imagesensors 314, such as inward-facing sensors, configured to capture animage of each eyeball of the user. Changes in the glints from the user'seyeballs and/or a location of a user's pupil as determined from imagedata gathered via the image sensor(s) 314 may be used to determine adirection of gaze. Further, a location at which gaze lines projectedfrom the user's eyes intersect the external display may be used todetermine an object at which the user is gazing (e.g. a displayedvirtual object and/or real background object). Gaze detection subsystem310 may have any suitable number and arrangement of light sources andimage sensors.

Display system 300 may further comprise additional sensors. For example,display system 300 may comprise a global positioning (GPS) subsystem 316to allow a location of the display system 300 to be determined. This mayallow a user of display system 300 to be presented different groups ofcommands associated with selected objects at different locations, forexample.

Display system 300 may further include one or more motion sensors 318 todetect movements of a user's head when the user is wearing displaysystem 300. Motion data may be used, potentially along with eye-trackingglint data and outward-facing image data, for gaze detection, as well asfor image stabilization to help correct for blur in images from theoutward-facing image sensor(s) 306. The use of motion data may allowchanges in gaze location to be tracked even if image data fromoutward-facing image sensor(s) 306 cannot be resolved. Likewise, motionsensors 318, as well as microphone(s) 308 and gaze detection subsystem310, also may be employed as user input devices, such that a user mayinteract with the display system 300 via gestures of the eye, neckand/or head, as well as via verbal commands. It will be understood thatsensors illustrated in FIGS. 2 and 3 are shown for the purpose ofexample and are not intended to be limiting in any manner, as any othersuitable sensors and/or combination of sensors may be utilized.

Display system 300 further comprises a controller 320 having a logicsubsystem 322 and a data holding subsystem 324 (which also may bereferred to as a storage system) in communication with the sensors, gazedetection subsystem 310, and display subsystem 304. Data holdingsubsystem 324 comprises instructions stored thereon that are executableby logic subsystem 322, for example, to receive and interpret inputsfrom the sensors, to identify movements of a user, to detect userselection of one or more objects, and to perform actions or commands onselected objects, among other tasks.

It will be appreciated that the depicted display devices 104 and 300 aredescribed for the purpose of example, and thus are not meant to belimiting. It is to be understood that the display device may includeadditional and/or alternative sensors, cameras, microphones, inputdevices, output devices, etc. than those shown without departing fromthe scope of this disclosure. Further, the physical configuration of adisplay device and its various sensors and subcomponents may take avariety of different forms without departing from the scope of thisdisclosure.

FIGS. 4-10 show example interactions of a user 106 wearing an augmentedreality display device 104 with persistent virtual objects relative toreal-world surfaces in an augmented reality environment. As remarkedabove, in some examples, a virtual object may be displayed as apersistent free-floating virtual object which follows a user's gazeabout an environment. During certain conditions, it may be desirable totransition a display of such a virtual object from a free-floating statewithin the environment to an attached state where the virtual object isdisplayed at least partially attached to a real-world surface in theenvironment. Various triggers may be used to transition a virtual objectfrom a free-floating state to an attached state.

For example, FIG. 4 shows a free-floating virtual object 130 at alocation 404 of a user focus 132 in an augmented reality environment.Free-floating virtual object 130 is a virtual object displayed to user106 via see-through display device 104 as unattached to any real-worldsurface in the augmented reality environment. For example, FIG. 4 showsfree-floating virtual object 130 at a location 404 within theenvironment as unattached or decoupled from real-world surface 118. Forexample, real-world surface 118 may be a wall, a glass window in a shop,a bill-board, etc.

Free-floating virtual object 130 at location 404 may be moveable withinthe augmented reality environment based on actions performed by user106. For example, user 106 may input a speech command such as “grab” andthen change a position of their focus 132 within the environment to movethe free-floating object to a new location. As another example, thefree-floating object may automatically move in response to a change inlocation of user focus 132 so that the free-floating virtual object isdisplayed at the point of focus or axis of gaze of the user, or at alocation referenced to the point of focus or axis of gaze of the user.

As remarked above, in some examples, during certain conditions, user 106may desire to attach a free-floating virtual object to a real-worldsurface within the environment in order to reduce intrusiveness of thevirtual object in the user's field of view. Further, attaching afree-floating virtual object to surfaces in an environment may assist auser in using their spatial memory to locate objects in the environment.For example, a user might place virtual objects at certain real-worldlocations throughout their home and locate the object based on the knownreal-world location. As a more specific example, a user may know thattheir virtual television may be found in a corner of the kitchen as theuser tends to look at that location while preparing dinner. Thus, thefree-floating virtual object 130 at location 404 may transition to anattached position, shown at 404′, on surface 118 in response to atrigger event. As one example, the trigger may comprise a speech inputcommand to attach the virtual object to a surface adjacent to thevirtual object. As another example, the free-floating virtual object maybe automatically attached to surface 118 when a distance 406 between thevirtual object and the surface is less than a threshold distance. Inthis way, the free-floating virtual object may appear to become“magnetized” to a proximate real-world surface and may automaticallysnap to the surface when the virtual object is moved by the user to athreshold distance from the surface. In some examples, the virtualobject's orientation may be modified to match an orientation of surface118 when the virtual object attaches to the surface. Modifying theobject's orientation to match an orientation of a surface to which it isattached may help to prevent the virtual object from appearing topenetrate the real-world surface, and thereby help to maintain theillusion of the virtual object as existing in the real worldenvironment.

With reference to FIG. 5, in some examples, once the virtual object 130is attached to the surface 118 at a location 404, the virtual object 130may move or slide across or along the surface 118 based on a gazelocation or position of focus of user 106. For example, FIG. 5 showsuser 106 with a first position of focus 502 at a first time at a firstlocation 404 along surface 118 where the virtual object 130 is attached.In response to a change in location of user focus or gaze, the attachedvirtual object may move along with the user focus while remainingattached to surface 118. For example, as shown in FIG. 5, at a secondtime following the first time, user 106 changes a location of user focusfrom the first position of focus 502 to a second, different, position offocus 504. In this example, the attached virtual object 130 moves frominitial location 404 to a new location 404′ so that the attached virtualobject remains at the location of user focus.

As shown in FIG. 6, in some examples, display of a virtual object 130attached to surface 118 may be adjusted in response to a change in anorientation, rotation, or location of user 106 while remaining in afixed location on surface 118. For example, the virtual object may bedisplayed as attached to surface 118 at a first orientation location 404to the user at a first position 602 with a first direction of focus 604substantially incident with surface 118. However, in response to achange in user position in the environment from the first position 602to a second position 606, the virtual object may be displayed asattached to surface 118 at a second orientation location 404′ so thatthe virtual object remains incident to the direction of focus 608 of theuser at the second position 606.

Turning to FIG. 7, as shown at 702, once a virtual object 130 isattached to the surface 118 at a location 404, the virtual object maymove or slide across the surface 118 based on changes in user gazedirection or position of focus.

For example, FIG. 7 shows user 106 with a first position of focus 706 ata first time at a first location 404 along surface 118 where the virtualobject 130 is attached. In response to a change in location of userfocus or gaze, the attached virtual object may move along with the userfocus while remaining attached to surface 118. For example, as shown inFIG. 7 at 702, at a second time following the first time, user 106changes a location of user focus from the first position of focus 706 toa second, different, position of focus 708 and the virtual object 130 inturn moves to a new location 404′ on surface 118 corresponding to thesecond position of focus 708.

As shown in FIG. 7, in some examples, the attached virtual object at thenew location 404′ may become “pinned” to the surface so that theattached virtual object 130 becomes fixed at location 404′ on surface118. For example, user 106 may move the virtual object to the newlocation 404′ and may provide input, e.g., a speech input, to pin thevirtual object to the surface at its current location. In this example,the user moves virtual object 130 from the first attached location 404to a new attached location at 404′ and then provides input to fixedlyattach the virtual object at location 404′. This pinning operatingcauses the object to freeze at its current location on the wall, nolonger following the user's view. As shown in FIG. 7 at 704, after thevirtual object is pinned to the location 404′ the virtual object will bemaintained at the fixed location 404′ even when the user's gazedirection changes. For example, as shown at 704, the virtual object 130remains at fixed location 404′ even when the location of user focusreturns to position 706.

Further, in some examples, pinned virtual object 130 shown at 404′ maybecome unpinned from surface 118. For example, user 106 may provideinput, e.g., speech input, to unpin the virtual object from its currentfixed position at 404′. As another example, unpinning an object may beperformed in response to an action performed by user 106. For example,user 106 may attempt to nudge virtual object 130 away from its currentfixed position by focusing on the object and changing their point offocus away from the fixed position. If this change of user focus isgreater than a threshold amount then the object may become unpinned.After, the virtual object is unpinned from the surface, the virtualobject may again move along with changes in user focus while remainingattached to surface 118.

Real-world surfaces to which virtual objects may be attached may includevarious discontinuities which impede movement of a virtual object alongthe surface. Examples of discontinuities include corners, doors,windows, other objects attached to a real-world surface, etc. FIGS. 8-10show various examples of moving a virtual object attached to areal-world surface 118 past a discontinuity 124 in the surface.

For example, FIG. 8 shows a jump or snap operation where the virtualobject 130 is shown attached to the surface 118 with a discontinuity 124at a first location 404 and then jumps or snaps past the discontinuity124 to a second location 404′. For example, surface 118 shown in FIG. 8may include a first real-world surface 802 interfacing with a secondreal-world surface 804 where the discontinuity 124 is at the interfaceof surface 802 with surface 804. A position of user focus 806 mayinitially be on surface 802 at location 404 where the virtual object 130is initially attached. However, as the user changes the position offocus from focus 806 to focus 808 past the discontinuity 124 and ontosurface 804, the virtual object snaps from location 404 to new location404′ on surface 804 past the discontinuity. For example, in response toan angle between a direction of user focus and the second real-worldsurface 804 meeting a threshold condition, the virtual object 130 may bedisplayed on the second real-world surface 804 and not displayed on thefirst real-world surface 802.

As another example, FIG. 9 shows a bend or deformation operationperformed on the virtual object 130 as it moves across a discontinuity124 in surface 118. For example, surface 118 shown in FIG. 9 may includea first real-world surface 802 interfacing with a second real-worldsurface 804 where the discontinuity 124 is at the interface of surface802 with surface 804. A position of user focus 902 may initially be at alocation 404 on surface 804 where the virtual object 130 is attached.However, as the user changes the position of focus from focus 902 tofocus 904 at the discontinuity 124, the virtual object bends or deformsto accommodate the shape of the surface at the discontinuity so that afirst portion of the virtual object is displayed on the first real-worldsurface 802 adjacent to the discontinuity and a second portion of thevirtual object 130 is displayed on the second real-world surface 804adjacent to the discontinuity. As the user continues to adjust theirlocation of focus past the discontinuity toward surface 804, the virtualobject continues to bend around the discontinuity so that the amount ofthe virtual object contained in the first portion displayed on secondsurface 804 increases until the entire virtual object is transitionedacross the discontinuity to second surface 804.

As still another example, FIG. 10 shows a bridge or detachment operationperformed on a virtual object 130 to move past a discontinuity 124 insurface 118. In this example, the user focus 1002 is initially at alocation 404 on surface 118 where the virtual object is displayed asattached to surface 118. However, as the location of user focus changesfrom focus 1002 to focus 1004 at the discontinuity, the virtual objectis detached from the real-world surface 118 at the discontinuity andthen reattached at a location past the discontinuity. For example, asthe user gaze approaches the discontinuity 124 by a threshold amount,the virtual object may be displayed as detached from surface 118 or mayappear to bridge the discontinuity to move past it. It will beunderstood that the embodiments of FIGS. 8-10 are presented for thepurpose of example and are not intended to be limiting in any manner, asa virtual object attached to a physical surface may be configured tohandle a discontinuity of the physical surface in any other suitablemanner.

FIG. 11 shows an example method 1100 for operating a user interface onan augmented reality computing device to manage attachment operationsfor persistent virtual objects in an augmented reality environment. Forexample, as described above, during certain conditions, virtual objectsmay be attached or detached from various real-world surfaces in theenvironment based on policies associated with the virtual object.

At 1102, method 1100 includes displaying a virtual object via thesee-through display system as free-floating. For example, as shown inFIG. 1A, a virtual object 130 may be displayed via see-through displaydevice as free-floating in environment 100.

At 1104, method 1100 includes detecting a trigger to display the virtualobject as attached to a real world surface. For example, at 1106, atrigger may include user input, e.g., speech input, and method 1100 mayinclude receiving user input to trigger an attachment of the virtualobject to a real-world surface. As another example, the trigger maycomprise a threshold distance between the virtual object or user and thereal-world surface. Thus, at 1108, method 1100 may include determining aproximity of the virtual object or the user with the real world surface.

At 1110, method 1100 includes, in response to the trigger, displayingthe virtual object as attached to the real-world surface via thesee-through display system. Further, in some examples, the virtualobject may be displayed as attached to the real-world surface based onpolicies associated with the virtual object. For example, one or morepolicies associated with the virtual object may be obtained, where theone or more policies dictate one or more of which real-world surfacesthe virtual object may be attached to and under what conditions thevirtual object may be attached to a real-world surface. The virtualobject may be displayed on the see-through display system based on theone or more policies associated with the virtual object.

As remarked above, display of the virtual object as attached to thesurface may be based on a location of user focus. For example, at 1112,method 1100 may include detecting a location of user focus on thereal-world surface, e.g., via one or more sensors in display device 104.At 1114, method 1100 may include displaying the virtual object at thelocation of user focus on the real-world surface.

Further, at 1116, method 1100 may include displaying the virtual objectas pinned to the surface. For example, the virtual object may bedisplayed on the see-through display system at a fixed location on thereal-world surface that does not change based upon changes in thelocation of user focus. As another example, at 1118, method 1100 mayinclude displaying the virtual object as unpinned to the surface so thatthe object can be moved about the surface while remaining attached tothe surface.

It will be understood that, at any time, a user may choose to pin orunpin a virtual object relative to a surface to which it is attached.Thus, at 1120, method 1100 includes receiving user input pinning orunpinning the virtual object relative to the real-world surface. Pinningand unpinning may occur in any suitable manner. For example, asdescribed above, in some embodiments the user may provide input, e.g.,speech input, to pin or unpin the virtual object attached to thereal-world surface.

At 1122, method 1100 includes detecting a change in location of userfocus. For example, after displaying the virtual object as attached tothe real-world surface via the see-through display system, a change inthe location of user focus on the real-world surface may be detected. At1124, method 1100 includes determining if the virtual object is pinnedat a fixed location on the real-world surface. If the virtual object ispinned at a fixed location on the real-world surface at 1124, thenmethod proceeds to 1126. At 1126, method 1100 includes maintaining thevirtual object at a fixed location on the real-world surface.

However, if the virtual object is not pinned at a fixed location on thereal-world surface at 1124, then method 1100 proceeds to 1128. At 1128,method 1100 includes displaying the virtual object attached to thesurface at the new location of user focus. For example, in an unpinnedstate, a virtual object may move in response to changes in user focus ormay be updated based on a change in a location of a gaze of a user sothat the object remains in direct view of the user.

As mentioned above, a physical surface to which a virtual object isattached may comprise a discontinuity, such as a corner, an opening(e.g. a door), etc. Thus, at 1130, method 1100 optionally includesdetecting a change of user focus at a location of discontinuity in thereal-world surface. In some examples, this may include detecting adiscontinuity between a first real-world surface and a second real-worldsurface. In response, at 1132, method 1100 includes moving the virtualobject past the discontinuity in the real-world surface. For example, at1134 the object may jump or snap to move past the discontinuity. Asanother example, at 1136, the object may bend or deform to move past thediscontinuity. As still another example, at 1138, the object may bridgeor detach from the surface to move past the discontinuity. For example,the virtual object may detach from the first real-world surface at thediscontinuity and then reattach at a location past the discontinuity.

At 1140, method 1100 includes detecting a trigger to detach the virtualobject from the real-world surface. For example, the trigger may includea user input, e.g., speech input, and/or a user action or gesture. At1142, method 1100 includes, in response to the trigger, detaching thevirtual object from the real-world surface and displaying the virtualobject as free-floating.

In some embodiments, the methods and processes described above may betied to a computing system of one or more computing devices. Inparticular, such methods and processes may be implemented as acomputer-application program or service, an application-programminginterface (API), a library, and/or other computer-program product.

FIG. 12 schematically shows a non-limiting embodiment of a computingsystem 1200 that can enact one or more of the methods and processesdescribed above. Display device 104 may be one non-limiting example ofcomputing system 1200. Computing system 1200 is shown in simplifiedform. It will be understood that virtually any computer architecture maybe used without departing from the scope of this disclosure. Indifferent embodiments, computing system 1200 may take the form of adisplay device, wearable computing device, mainframe computer, servercomputer, desktop computer, laptop computer, tablet computer,home-entertainment computer, network computing device, gaming device,mobile computing device, mobile communication device (e.g., smartphone), etc.

Computing system 1200 includes a logic subsystem 1202 and a storagesubsystem 1204. Computing system 1200 may optionally include a displaysubsystem 1206, input subsystem 1208, communication subsystem 1210,and/or other components not shown in FIG. 12.

Logic subsystem 1202 includes one or more physical devices configured toexecute instructions. For example, the logic subsystem may be configuredto execute instructions that are part of one or more applications,services, programs, routines, libraries, objects, components, datastructures, or other logical constructs. Such instructions may beimplemented to perform a task, implement a data type, transform thestate of one or more components, or otherwise arrive at a desiredresult.

The logic subsystem may include one or more processors configured toexecute software instructions. Additionally or alternatively, the logicsubsystem may include one or more hardware or firmware logic machinesconfigured to execute hardware or firmware instructions. The processorsof the logic subsystem may be single-core or multi-core, and theprograms executed thereon may be configured for sequential, parallel ordistributed processing. The logic subsystem may optionally includeindividual components that are distributed among two or more devices,which can be remotely located and/or configured for coordinatedprocessing. Aspects of the logic subsystem may be virtualized andexecuted by remotely accessible, networked computing devices configuredin a cloud-computing configuration.

Storage subsystem 1204 includes one or more physical devices configuredto hold data and/or instructions executable by the logic subsystem toimplement the methods and processes described herein. When such methodsand processes are implemented, the state of storage subsystem 1204 maybe transformed—e.g., to hold different data.

Storage subsystem 1204 may include removable media and/or built-indevices. Storage subsystem 1204 may include optical memory devices(e.g., CD, DVD, HD-DVD, Blu-Ray Disc, etc.), semiconductor memorydevices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices(e.g., hard-disk drive, floppy-disk drive, tape drive, MRAM, etc.),among others. Storage subsystem 1204 may include volatile, nonvolatile,dynamic, static, read/write, read-only, random-access,sequential-access, location-addressable, file-addressable, and/orcontent-addressable devices.

It will be appreciated that storage subsystem 1204 includes one or morephysical devices and excludes propagating signals per se. However, insome embodiments, aspects of the instructions described herein may bepropagated by a pure signal (e.g., an electromagnetic signal, an opticalsignal, etc.) via a communications medium, as opposed to being stored ona storage device. Furthermore, data and/or other forms of informationpertaining to the present disclosure may be propagated by a pure signal.

In some embodiments, aspects of logic subsystem 1202 and of storagesubsystem 1204 may be integrated together into one or morehardware-logic components through which the functionally describedherein may be enacted. Such hardware-logic components may includefield-programmable gate arrays (FPGAs), program- andapplication-specific integrated circuits (PASIC/ASICs), program- andapplication-specific standard products (PSSP/ASSPs), system-on-a-chip(SOC) systems, and complex programmable logic devices (CPLDs), forexample.

When included, display subsystem 1206 may be used to present a visualrepresentation of data held by storage subsystem 1204. This visualrepresentation may take the form of a graphical user interface (GUI). Asthe herein described methods and processes change the data held by thestorage subsystem, and thus transform the state of the storagesubsystem, the state of display subsystem 1206 may likewise betransformed to visually represent changes in the underlying data.Display subsystem 1206 may include one or more display devices utilizingvirtually any type of technology. Such display devices may be combinedwith logic subsystem 1202 and/or storage subsystem 1204 in a sharedenclosure, or such display devices may be peripheral display devices.

When included, input subsystem 1208 may comprise or interface with oneor more user-input devices such as a keyboard, mouse, touch screen, orgame controller. In some embodiments, the input subsystem may compriseor interface with selected natural user input (NUI) componentry. Suchcomponentry may be integrated or peripheral, and the transduction and/orprocessing of input actions may be handled on- or off-board. Example NUIcomponentry may include a microphone for speech and/or voicerecognition; an infrared, color, stereoscopic, and/or depth camera formachine vision and/or gesture recognition; a head tracker, eye tracker,accelerometer, and/or gyroscope for motion detection and/or intentrecognition; as well as electric-field sensing componentry for assessingbrain activity.

When included, communication subsystem 1210 may be configured tocommunicatively couple computing system 1200 with one or more othercomputing devices. Communication subsystem 1210 may include wired and/orwireless communication devices compatible with one or more differentcommunication protocols. As non-limiting examples, the communicationsubsystem may be configured for communication via a wireless telephonenetwork, or a wired or wireless local- or wide-area network. In someembodiments, the communication subsystem may allow computing system 1200to send and/or receive messages to and/or from other devices via anetwork such as the Internet.

It will be understood that the configurations and/or approachesdescribed herein are exemplary in nature, and that these specificembodiments or examples are not to be considered in a limiting sense,because numerous variations are possible. The specific routines ormethods described herein may represent one or more of any number ofprocessing strategies. As such, various acts illustrated and/ordescribed may be performed in the sequence illustrated and/or described,in other sequences, in parallel, or omitted. Likewise, the order of theabove-described processes may be changed.

The subject matter of the present disclosure includes all novel andnon-obvious combinations and sub-combinations of the various processes,systems and configurations, and other features, functions, acts, and/orproperties disclosed herein, as well as any and all equivalents thereof.

1. On an augmented reality computing device comprising a display system,a method of operating a user interface, the method comprising:displaying a virtual object via the display system as free-floating;identifying a surface via image data acquired with the augmented realitycomputing device; comparing the surface to a policy associated with thevirtual object; and based at least on comparing the surface to thepolicy associated with the virtual object, displaying the virtual objectas attached to the surface.
 2. The method of claim 1, wherein thesurface is a first surface, further comprising identifying a secondsurface via the image data, comparing the second surface to the policyassociated with the virtual object, and not displaying the virtualobject as attached to the second surface based upon comparing the secondsurface to the policy associated with the virtual object.
 3. The methodof claim 1, wherein the surface comprises a real-world surface.
 4. Themethod of claim 1, wherein the trigger comprises a threshold distancebetween the virtual object and the surface.
 5. The method of claim 1,further comprising updating displaying of the virtual object on thedisplay system based on a change in a location of a gaze of a user. 6.The method of claim 1, further comprising: detecting a trigger to detachthe virtual object from the surface; and in response to the trigger todetach the virtual object from the surface, detaching the virtual objectfrom the surface and displaying the virtual object as free-floating. 7.The method of claim 1, further comprising: detecting a location of userfocus on the surface; and displaying the virtual object on the surfaceat a location based upon the location of user focus.
 8. The method ofclaim 7, wherein the surface is a first surface, and further comprisingafter displaying the virtual object as attached to the first surface viathe display system, detecting a change in the location of user focus onthe first surface and maintaining the virtual object attached at a fixedlocation on the first surface.
 9. The method of claim 8, furthercomprising detecting a discontinuity on the first surface at a locationof user focus, and detaching the virtual object from the first surfaceat the discontinuity and then reattaching the virtual object at alocation past the discontinuity.
 10. The method of claim 8, furthercomprising detecting a discontinuity between the first surface and asecond surface, and displaying a first portion of the virtual object onthe first surface adjacent to the discontinuity and a second portion ofthe virtual object on the second surface adjacent to the discontinuity.11. The method of claim 8, further comprising detecting a discontinuitybetween the first surface and a second surface, and in response to anangle between a direction of user focus and the second surface meeting athreshold condition, displaying the virtual object on the second surfaceand not displaying the virtual object on the first surface.
 12. Themethod of claim 1, wherein the policy specifies one or more of a colorand a texture of a surface to which the virtual object can attach. 13.The method of claim 1, wherein the policy specifies one or more of size,a location, and an orientation of a surface to which the virtual objectcan attach.
 14. On an augmented reality computing device comprising asee-through display system, a method for operating a user interface, themethod comprising: displaying a virtual object via the display system asfree-floating; identifying a surface of a first orientation and asurface of a second, different orientation via image data acquired withthe augmented reality computing device; based at least on comparing thesurface to a policy associated with the virtual object, displaying thevirtual object as attached to the surface of the first orientation whenthe virtual object is closer than a threshold distance from the surfaceof the first orientation, and not displaying the virtual object asattached to the surface of the second orientation when the virtualobject is closer than the threshold distance from the surface of thesecond orientation.
 15. The method of claim 14, wherein the policyspecifies one or more of which real-world surfaces the virtual objectmay be attached to and under what conditions the virtual object may beattached to a real-world surface, and displaying the virtual object onthe see-through display system based on the one or more policiesassociated with the virtual object.
 16. The method of claim 14, furthercomprising, after displaying the virtual object attached at the fixedlocation on the first real-world surface on the see-through displaysystem, detecting a second change in location of user focus on the firstreal-world surface and maintaining the virtual object attached at thefixed location on the first real-world surface.
 17. The method of claim14, further comprising: detecting a trigger to detach the virtual objectfrom the first real-world surface; and in response to the trigger todetach the virtual object from the first real-world surface, detachingthe virtual object from the first real-world surface and displaying thevirtual object as free-floating.
 18. A wearable augmented realitycomputing device, comprising: a see-through display device; a sensorsystem including one or more sensors; a logic subsystem; and a storagesubsystem comprising instructions stored thereon that are executable bythe logic subsystem to identify a surface via image data acquired withthe augmented reality computing device; compare the surface to a policyassociated with the virtual object; and based at least on comparing thesurface to the policy associated with the virtual object, display thevirtual object as attached to the surface.
 19. The device of claim 18,wherein the policy specifies one or more of a color and a texture of asurface to which the virtual object can attach.
 20. The device of claim18, wherein the policy specifies one or more of size, a location, and anorientation of a surface to which the virtual object can attach.